The Joy of SAX

Most current XQuery implementations require that all XML data reside in memory in one form or another before they start processing the data. This is unacceptable for large XML documents. The only event-based techniques that do not require the materialization of all data in memory are based on transducers. Transducers use SAX events to evaluate XPath queries using finite state machines. SAX allows parsing and processing of XML documents in a stream fashion, but is hard to use even for simple tasks since it leaves the burden to the programmer to maintain and propagate the state between events. As is well known to people who have written non-trivial SAX event handlers, SAX can be used effectively under a disciplined use of pipes made out of SAX event handlers, where a producer object sends SAX events to a consumer object, which in turn becomes the producer for the next consumer, etc, thus forming a pipe. In this paper, I present an event-based XQuery interpreter, called XStreamQuery, based entirely on SAX pipelines. These pipelines consist of pipes that not only push SAX events from producers to consumers but also return feedback to the producers that allows them to cut the pipes short. This immediate feedback is difficult to achieve with transducers. Even though XStreamQuery is still a prototype system, I expect that, for simple XPath queries, it will have the same low memory overhead as transducers, although with a bit more computational overhead. But I also expect XStreamQuery to beat transducers for complex XQueries, especially for XPaths that contain predicates with embedded XQueries and for deeply-nested XQueries. More importantly, XStreamQuery allows the compositional generation of query plans, eg, one XPath step at a time, rather than requiring to look at the entire path to generate code. This makes the query plan generation very easy. It also allows path pipelines to be seamlessly integrated with the rest of XQuery. In fact, for a substantial subset of XQuery (without optimization), the plan generation code is less than 200 lines of Java while the entire XQuery processor is less than 2000 lines of Java code. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission of the authors. Informal Proceedings of the First International Workshop on XQuery Implementation, Experience, and Perspectives (XIME-P), June 17-18, 2004, Paris, France.